Input Sequence Research Articles

Transient surrogate modeling of modally reduced structures with discontinuous loads and damping

AbstractIn this work, a surrogate model for structural, transient and discontinuously excited finite element method simulations is developed. This allows to reduce the computational effort of repeated calculations of identical models under different load cases. The architecture of the surrogate combines fully connected neural network layers with long short-term memory layers. For the reproduction of different damping ratios, a categorical variable is added to the continuous input data. Based on a recursive flow of the predicted data back to the input layer, long-term dependencies do not vanish due to short-input sequences. The system dimension is reduced by applying the model-order reduction technique for modal decomposition. The high accuracy of the surrogate and the reduction of computational costs are shown on an academic example of a cantilever beam and a real-world example of a robot. The advantages of our approach are illustrated in comparison with state-of-the-art surrogates for transient finite element analysis. By using the surrogate proposed in this study, oscillations due to discontinuous excitation of mechanical structures can be reproduced. For this purpose, only short-input sequences are necessary since the excitation of the oscillations does not have to be part of the input sequence during the whole duration of the oscillations. Due to the categorical variable for the damping ratio, the surrogate can account for the influence of different damping in parameter studies.

ConsensusPrime—A Bioinformatic Pipeline for Efficient Consensus Primer Design—Detection of Various Resistance and Virulence Factors in MRSA—A Case Study

Background: The effectiveness and reliability of diagnostic tests that detect DNA sequences largely hinge on the quality of the used primers and probes. This importance is especially evident when considering the specific sample being analyzed, as it affects the molecular background and potential for cross-reactivity, ultimately determining the test’s performance. Methods: Predicting primers based on the consensus sequence of the target has multiple advantages, including high specificity, diagnostic reliability, broad applicability, and long-term validity. Automated curation of the input sequences ensures high-quality primers and probes. Results: Here, we present a use case for developing a set of consensus primers and probes to identify antibiotic resistance and virulence genes in Staphylococcus (S.) aureus using the ConsensusPrime pipeline. Extensive qPCR experiments with several S. aureus strains confirm the exceptional quality of the primers designed using the pipeline. Conclusions: By improving the quality of the input sequences and using the consensus sequence as a basis, the ConsensusPrime pipeline pipeline ensures high-quality primers and probes, which should be the basis of molecular assays.

An efficient CNN-transformer hybrid approach for water turbine unit failure prediction

Water Turbine Unit is the core equipment of water power generation. It is the most important research object in hydroelectric power. The analysis of daily monitoring data of Water Turbines can evaluate its running state and avoid the loss caused by failure. In this paper, A hybrid neural network architecture which CNN combines Transformer is proposed to evaluate the health state of water turbines based on multivariate long time series data. It has two core components: (i) dilated convolution applies to capture low-level and local semantic information, then the output of convolutional layers is divided into subseries-level patches by time, these patches are regarded as input tokens of Transformer layers; (ii) utilizing self-attention of Transformer to extract high-level and global semantic information. Patching design naturally has benefit as follow: (i) local semantic information is retained in Transformer input tokens and high-level semantic confirm to common sense of human understanding through low-level construction; (ii) the length of Transformer input sequence is greatly shorted and attention can be more concentrated compared with point-wise form. Meanwhile, the computation and memory usage reduces at the same time. The experimental result indicates that the hybrid architecture can achieves excellent performance in time series understanding.

Short-term multi-step-ahead sector-based traffic flow prediction based on the attention-enhanced graph convolutional LSTM network (AGC-LSTM)

AbstractAccurate sector-based air traffic flow predictions are essential for ensuring the safety and efficiency of the air traffic management (ATM) system. However, due to the inherent spatial and temporal dependencies of air traffic flow, it is still a challenging problem. To solve this problem, some methods are proposed considering the relationship between sectors, while the complicated spatiotemporal dynamics and interdependencies between traffic flow of route segments related to the sector are not taken into account. To address this challenge, the attention-enhanced graph convolutional long short-term memory network (AGC-LSTM) model is applied to improve the short-term sector-based traffic flow prediction, in which spatial structures of route segments related to the sector are considered for the first time. Specifically, the graph convolutional networks (GCN)-LSTM network model was employed to capture spatiotemporal dependencies of the flight data, and the attention mechanism is designed to concentrate on the informative features from key nodes at each layer of the AGC-LSTM model. The proposed model is evaluated through a case study of the typical enroute sector in the central–southern region of China. The prediction results show that MAE reduces by 14.4% compared to the best performing GCN-LSTM model among the other five models. Furthermore, the study involves comparative analyses to assess the influence of route segment range, input and output sequence lengths, and time granularities on prediction performance. This study helps air traffic managers predict flight situations more accurately and avoid implementing overly conservative or excessively aggressive flow management measures for the sectors.

Development of a Screen Keyboard System with Radially‐Arranged Keys and its Effectiveness for Typing Including Eye‐Gaze Input

AbstractThis study aims to develop an effective screen‐keyboard system from a new perspective of key arrangements. In the proposed screen keyboard system, keys are centralized by arranging them radially, with alphabetical keys arranged based on the articulation for speech production. A gesture input method and click input method are implemented considering the mora‐based Japanese sound system with syllables containing a consonant‐vowel (CV) structure. In a preliminary experiment, the cumulative pixel distances required to press keys for the input sequences were evaluated as a simulation to confirm the basic performance of the keyboard system. The Euclidean pixel distances between the keys on the keyboard images were accumulated and compared between the proposed keyboard system and a typical QWERTY keyboard. Typing experiments involving participants were performed to confirm the effectiveness of the proposed keyboard system. The experimental results reveal that the proposed screen keyboard system enables users to generate efficient typing performances using a computer mouse and eye gaze via short‐term training. © 2024 Institute of Electrical Engineer of Japan and Wiley Periodicals LLC.

AFMF: Time Series Anomaly Detection Framework with Modified Forecasting

Forecasting-based method is one of prevalent unsupervised time series anomaly detection approaches. Currently, large portions of existing forecasting-based methods are devoted to discussing the feature extraction of input sequences and targeting at accurate predictions. In essence, their frameworks and core ideas are identical to the pure forecasting models. However, the distinctiveness of anomalies is affected by not only forecasting accuracy, but also many other factors. This paper summarizes three other dominant factors: (1) Scale disparity; (2) Discrete variate; (3) Input anomaly. They are common and non-negligible in real-world anomaly detection. Moreover, we propose AFMF: a time series Anomaly detection Framework with Modified Forecasting to solve them respectively by its three key components, i.e., Local Instance Normalization, Lopsided Forecasting and Progressive Adjacent Masking. The first two are refined descendants of existing mechanisms while the third component is completely novel. Extensive experiments on ten benchmarks verify that AFMF can be combined with any forecasting or forecasting-based anomaly detection method to achieve SOTA anomaly detection performances. The source code is available at https://github.com/OrigamiSL/AFMF.

Two-pathway spatiotemporal representation learning for extreme water temperature prediction

Accurate predictions of extreme water temperatures are criticalto understanding the variability of the marine environment and reducing marine disasters maximized by global warming. In this study, we propose a two-pathway framework with separated spatial and temporal encoders for accurate prediction of water temperature, especially extremely high water temperature, through effective spatiotemporal representation learning. The spatial and temporal encoder networks based on the Transformer’s self-attention mechanism performs the task of predicting the water temperature time series at the 16 coastal locations around the Korean Peninsula for the seven consecutive days ahead at daily intervals with various combinations of patch embedding methods, positional embedding for for spatial features. Comparative experiments with conventional deep convolutional and recurrent networks are also conducted for comparison. By comparing and assessing these results, the proposed two-pathway framework can improve the predictability of extremely high coastal water temperature by better capturing spatiotemporal interrelationships and long-range dependencies from open ocean and regional sea, and further determines the optimal architectural details of self-attention-based spatial and temporal encoders. Furthermore, to examine the explainability of the proposed model and its consistency with domain knowledge, spatial and temporal attention maps are visualized and analyzed that represents weights for spatiotemporal input sequences that are more relevant to predict for future predictions.

Retrieval for Extremely Long Queries and Documents with RPRS: A Highly Efficient and Effective Transformer-based Re-Ranker

Retrieval with extremely long queries and documents is a well-known and challenging task in information retrieval and is commonly known as Query-by-Document (QBD) retrieval. Specifically designed Transformer models that can handle long input sequences have not shown high effectiveness in QBD tasks in previous work. We propose a Re-Ranker based on the novel Proportional Relevance Score (RPRS) to compute the relevance score between a query and the top- k candidate documents. Our extensive evaluation shows RPRS obtains significantly better results than the state-of-the-art models on five different datasets. Furthermore, RPRS is highly efficient, since all documents can be pre-processed, embedded, and indexed before query time that gives our re-ranker the advantage of having a complexity of O(N) , where N is the total number of sentences in the query and candidate documents. Furthermore, our method solves the problem of the low-resource training in QBD retrieval tasks as it does not need large amounts of training data and has only three parameters with a limited range that can be optimized with a grid search even if a small amount of labeled data is available. Our detailed analysis shows that RPRS benefits from covering the full length of candidate documents and queries.

Peringkasan Teks Otomatis Abstraktif Menggunakan Transformer Pada Teks Bahasa Indonesia

Abstractive text summarization is used to generate summaries that are similar to human-written summaries. To achieve that capability, recurrent deep learning architectures were usually applied, such as RNN, LSTM, and GRU. In previous studies about abstractive text summarization in Indonesian, recurrent models were widely used and there were cohesion and grammar errors that appeared in the generated summaries—this could have an impact on performance. Currently, there is Transformer, a relatively new architecture that relies on the attention mechanism entirely. Due to its non-recurrent nature, Transformer overcomes the problem of dependency the on hidden states that occurs in recurrent models and it can retain information on all input sequences. In this study, we use Transformer to evaluate how good it is at abstractive text summarization in Indonesian. The training was conducted using the pre-trained T5 model with IndoSum dataset which contains around 19K news-summary pairs. We achieved evaluation scores of 0.61 ROUGE-1 and 0.51 ROUGE-2.

Spatial–temporal combination and multi-head flow-attention network for traffic flow prediction

Traffic flow prediction based on spatial–temporal data plays a vital role in traffic management. However, it still faces serious challenges due to the complex spatial–temporal correlation in nonlinear spatial–temporal data. Some previous methods have limited ability to capture spatial–temporal correlation, and ignore the quadratic complexity problem in the traditional attention mechanism. To this end, we propose a novel spatial–temporal combination and multi-head flow-attention network (STCMFA) to model the spatial–temporal correlation in road networks. Firstly, we design a temporal sequence multi-head flow attention (TS-MFA), in which the unique source competition mechanism and sink allocation mechanism make the model avoid attention degradation without being affected by inductive biases. Secondly, we use GRU instead of the linear layer in traditional attention to map the input sequence, which further enhances the temporal modeling ability of the model. Finally, we combine the GCN with the TS-MFA module to capture the spatial–temporal correlation, and introduce residual mechanism and feature aggregation strategy to further improve the performance of STCMFA. Extensive experiments on four real-world traffic datasets show that our model has excellent performance and is always significantly better than other baselines.

LMCrot: an enhanced protein crotonylation site predictor by leveraging an interpretable window-level embedding from a transformer-based protein language model.

Recent advancements in natural language processing have highlighted the effectiveness of global contextualized representations from protein language models (pLMs) in numerous downstream tasks. Nonetheless, strategies to encode the site-of-interest leveraging pLMs for per-residue prediction tasks, such as crotonylation (Kcr) prediction, remain largely uncharted. Herein, we adopt a range of approaches for utilizing pLMs by experimenting with different input sequence types (full-length protein sequence versus window sequence), assessing the implications of utilizing per-residue embedding of the site-of-interest as well as embeddings of window residues centered around it. Building upon these insights, we developed a novel residual ConvBiLSTM network designed to process window-level embeddings of the site-of-interest generated by the ProtT5-XL-UniRef50 pLM using full-length sequences as input. This model, termed T5ResConvBiLSTM, surpasses existing state-of-the-art Kcr predictors in performance across three diverse datasets. To validate our approach of utilizing full sequence-based window-level embeddings, we also delved into the interpretability of ProtT5-derived embedding tensors in two ways: firstly, by scrutinizing the attention weights obtained from the transformer's encoder block; and secondly, by computing SHAP values for these tensors, providing a model-agnostic interpretation of the prediction results. Additionally, we enhance the latent representation of ProtT5 by incorporating two additional local representations, one derived from amino acid properties and the other from supervised embedding layer, through an intermediate fusion stacked generalization approach, using an n-mer window sequence (or, peptide/fragment). The resultant stacked model, dubbed LMCrot, exhibits a more pronounced improvement in predictive performance across the tested datasets. LMCrot is publicly available at https://github.com/KCLabMTU/LMCrot.

A Joint Time-Frequency Domain Transformer for multivariate time series forecasting

In order to enhance the performance of Transformer models for long-term multivariate forecasting while minimizing computational demands, this paper introduces the Joint Time-Frequency Domain Transformer (JTFT). JTFT combines time and frequency domain representations to make predictions. The frequency domain representation efficiently extracts multi-scale dependencies while maintaining sparsity by utilizing a small number of learnable frequencies. Simultaneously, the time domain (TD) representation is derived from a fixed number of the most recent data points, strengthening the modeling of local relationships and mitigating the effects of non-stationarity. Importantly, the length of the representation remains independent of the input sequence length, enabling JTFT to achieve linear computational complexity. Furthermore, a low-rank attention layer is proposed to efficiently capture cross-dimensional dependencies, thus preventing performance degradation resulting from the entanglement of temporal and channel-wise modeling. Experimental results on eight real-world datasets demonstrate that JTFT outperforms state-of-the-art baselines in predictive performance.

Wind-Speed Multi-Step Forecasting Based on Variational Mode Decomposition, Temporal Convolutional Network, and Transformer Model

Reliable and accurate wind-speed forecasts significantly impact the efficiency of wind power utilization and the safety of power systems. In addressing the performance enhancement of transformer models in short-term wind-speed forecasting, a multi-step prediction model based on variational mode decomposition (VMD), temporal convolutional network (TCN), and a transformer is proposed. Initially, the Dung Beetle Optimizer (DBO) is utilized to optimize VMD for decomposing non-stationary wind-speed series data. Subsequently, the TCN is used to extract features from the input sequences. Finally, the processed data are fed into the transformer model for prediction. The effectiveness of this model is validated by comparison with six other prediction models across three datasets, demonstrating its superior accuracy in short-term wind-speed forecasting. Experimental findings from three distinct datasets reveal that the developed model achieves an average improvement of 52.1% for R2. To the best of our knowledge, this places our model at the leading edge of wind-speed prediction for 8 h and 12 h forecasts, demonstrating MSEs of 1.003 and 0.895, MAEs of 0.754 and 0.665, and RMSEs of 1.001 and 0.946, respectively. Therefore, this research offers significant contributions through a new framework and demonstrates the utility of the transformer in effectively predicting short-term wind speed.

A Meta heuristic based deep learning classifier for effective dengue disease prediction in IoT‐Fog system

AbstractThe primary aim of this article is to propose an effective dengue disease monitoring system by integrating deep learning models with Internet of Things (IoT) and fog computing. In this context, the disease related parameters are collected using IoT devices and further, the reports are securely transmitted to the healthcare facilities utilizing fog computing. In this article, the misdiagnosis error is reduced utilizing attention based long short term memory (ALSTM) network with coati optimization algorithm (COA). The Attention model allocates higher weight to significant parameters and pay additional consideration while training the model. Therefore, the conventional LSTM network is combined with a self‐attention mechanism for enabling this model to focus on relevant parts of the input sequences for precise dengue disease prediction. The Attention Model differs from the conventional model since it delivers higher amount of data into the decoder and improves additional stages to the attention decoder before making its output. Furthermore, the COA is integrated with the ALSTM network for optimal hyper‐parameter selection. The ALSTM‐COA model evaluates the dengue related parameters like nausea, vomiting, skin rash, joint pain, muscle pain, soft bleeding, temperature of water‐sites, humidity of water‐sites, and so forth, for timely dengue disease diagnosis and clinical decision‐making. The experiments performed on a real time dataset state that the ALSTM‐COA model achieves significant prediction results by utilizing different performance measures like recall, accuracy, fall‐out, miss rate, training time, and testing time. The ALSTM‐COA model obtains an accuracy of 99.27%, latency of 0.03 s, and time complexity of 8 s for 500 health records, which are superior to the comparative deep learning models.

An Attention-Based Deep Learning Approach for Lithium-ion Battery Lifespan Prediction: Analysis and Experimental Validation

The potential for lithium-ion batteries to become unstable can lead to operational malfunctions within the system and result in safety incidents. Therefore, accurately forecasting the remaining useful life (RUL) is beneficial in mitigating the likelihood of battery failure and prolonging its operational lifespan. Hence, precise estimation of RUL can help prevent numerous safety incidents and minimize resource wastage, presenting a significant and complex issue. This paper introduces a Deep Learning (DL) model that utilizes Long Short-Term Memory (LSTM) and attention mechanism to improve the accuracy of predicting the RUL of lithium-ion batteries. Initially, the battery capacity regeneration phenomenon is captured by applying four LSTM layers, followed by implementing an attention mechanism to align input and output sequences based on the content or semantics of the input sequence. Finally, the final prediction outcomes are generated via a Fully Connected (FC) layer. The efficacy of the proposed model is assessed through the utilization of the NASA dataset, and its performance is contrasted with various deep learning models to highlight its efficacy. Results from the experiments demonstrate that the suggested At-LSTM presents a robust option for forecasting the RUL of lithium-ion batteries, as it delivers superior results compared to all other models examined.

Vibration suppression of smart composite beam using model predictive controller

Abstract This work presents an adaptive model predictive control (MPC) strategy to suppress the vibration in a laminated composite beam. The control method incorporates a system identification algorithm to estimate the system parameters online, which provides a precise simulation of system dynamics. A fixed-free cantilever composite beam equipped with piezoelectric actuators was used to evaluate the efficacy of the control method. The sensors and actuators are securely bonded to the upper and lower surfaces at arbitrary locations along the beam’s length. A unified mechanical displacement field is applied to all layers, while displacements are considered independently for each layer. The beam is composed of eight layers of material, each with a thickness of 0.2 mm and orientations specified as (90°/0°/90°/0°). To achieve the best performance, the parameters of the MPC were adjusted numerically. The numerical analysis revealed that placing the actuator near the clamped end at the fixed end resulted in superior control outcomes, with a settling time of approximately 1.8 s. Conversely, the longest settling time occurred when the actuator was positioned at the free end, taking around 4 s. This model could potentially be expanded to address vibration in more intricate beams exhibiting nonlinear characteristics. The deflection readings measured at the end of the beam have been utilized as feedback control signals for predicting future behavior over a predetermined control horizon. The subsequent cost function is minimized through a quadratic equation to determine the sequence of optimal yet constrained control inputs. The suggested active vibration control system is then implemented and assessed numerically to examine the effectiveness of the control method.

Accurately clustering biological sequences in linear time by relatedness sorting

Clustering biological sequences into similar groups is an increasingly important task as the number of available sequences continues to grow exponentially. Search-based approaches to clustering scale super-linearly with the number of input sequences, making it impractical to cluster very large sets of sequences. Approaches to clustering sequences in linear time currently lack the accuracy of super-linear approaches. Here, I set out to develop and characterize a strategy for clustering with linear time complexity that retains the accuracy of less scalable approaches. The resulting algorithm, named Clusterize, sorts sequences by relatedness to linearize the clustering problem. Clusterize produces clusters with accuracy rivaling popular programs (CD-HIT, MMseqs2, and UCLUST) but exhibits linear asymptotic scalability. Clusterize generates higher accuracy and oftentimes much larger clusters than Linclust, a fast linear time clustering algorithm. I demonstrate the utility of Clusterize by accurately solving different clustering problems involving millions of nucleotide or protein sequences.

Improving respiratory signal prediction with a deep neural network and simple changes to the input and output data format

Objective. To improve respiratory gating accuracy and radiation treatment throughput, we developed a generalized model based on a deep neural network (DNN) for predicting any given patient’s respiratory motion. Approach. Our model uses long short-term memory (LSTM) based on a recurrent neural network (RNN), and improves upon common techniques. The first improvement is that the data input is not a one-dimensional sequence, but two-dimensional block data. This shortens the input sequence length, reducing computation time. Second, the output is not a scalar, but a sequence prediction. This increases the amount of available data, allowing improved prediction accuracy. For training and evaluation of our model, 434 sets of real-time position management data were retrospectively collected from clinical studies. The data were separated in a ratio of 4:1, with the larger set used for training models and the remaining set used for testing. We measured the accuracy of respiratory signal prediction and amplitude-based gating with prediction windows equaling 133, 333, and 533 ms. This new model was compared with the original LSTM and a non-recurrent DNN model. Main results. The mean absolute errors with the prediction window at 133, 333 and 533 ms were 0.036, 0.084, 0.119 with our model; 0.049, 0.14, 0.246 with the original LSTM-based model; and 0.041, 0.119, 0.16 with the non-recurrent DNN model, respectively. The computation time were 0.66 ms with our model; 0.63 ms the original LSTM-based model; 1.60 ms the non-recurrent DNN model, respectively. The accuracies of amplitude-based gating with the same prediction window settings and a duty cycle of approximately 50% were 98.3%, 95.8% and 92.7% with our model, 97.6%, 93.9% and 87.2% with the original LSTM-based model; and 97.9%, 94.3% and 89.5% with the non-recurrent DNN model, respectively. Significance. Our RNN algorithm for respiratory signal prediction successfully estimated tumor positions. We believe it will be useful in respiratory signal prediction technology.

ATTnet: An explainable gated recurrent unit neural network for high frequency electricity price forecasting

The primary contribution of this study is the proposal of an explainable deep-learning neural network (ATTnet) that employs an attention mechanism to achieve accurate electricity spot price forecasting and an explainable model pipeline. The concise, single-stream network consists of a 5-head attention mechanism and gated recurrent units, which have been developed to model the temporal dependencies of the volatile market data. In addition to introducing a novel neural network architecture for volatile time series data, this study makes a substantial contribution by investigating prediction factors in two ways: temporally via the attention scores from the input sequences and globally via feature Shapely values. In real-time electricity price prediction, historical prices, temperature, hour, and zonal load are found to be the most important variables. The deep learning model was tested on real-time price profiles from eight generators within the New York Independent System Operator (NYISO) network. The proposed model achieves performance gains of 21% in MAE and 22% in MAPE over the state-of-the-art benchmark methods.

State Estimation via Designing Controller and State Estimation-Based Stabilization for Boolean Control Networks.

This article investigates the issues of state estimation and state estimation-based stabilization for Boolean control networks (BCNs). Unlike previous state observers, this article proposes an optimal state estimator by designing a particular input sequence for the first time, where the maximum-minimum method is employed such that the state of BCNs can be uniquely estimated in short time steps. A minimum reconstructible state set (MRSS) is constructed to determine this input sequence. Next, based on the estimated state, a finite-time stabilization scheme is proposed by constructing a switching controller consisting of three stages. A controller is first developed to estimate the state of BCNs in finite-time steps, and a state reachable controller is also provided to make the state of BCNs reachable to a given equilibrium point. Subsequently, a constant controller is further developed to stabilize the state of BCNs to the equilibrium point. Finally, an oxidative stress response model is used to illustrate the effectiveness of the proposed results.